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Colorectal image classification is a novel application area in medical image 
processing. Colorectal images are one of the most prevalent malignant tumour 
disease type in the world. However, due to the complexity of histopathological 
imaging, the most accurate and effective classification still needs to be 
addressed. In this work we proposed a novel architecture of convolution 


neural network with deep learning models for the multiclass classification of 
histopathology images. We achieved the findings using three deep learning 
models, including the vgg16 with 96.16% and a modified version of Resnet50 
with 97.08%, however the proposed Adaptive Resnet152 model generated the 
best accuracy of 98.38%. The colorectal image multiclass dataset is publicly 
available which has 5000 images with 8 classes. In this study we have 
increased all classes equally, total 15000 images have been generated using 
image augmentation technique. This dataset consists of 60% training images 
and 40% testing images. The suggested method in this paper produced better 
results than the existing histopathology image categorization methods with 
the lowest error rate. For histopathological image categorization, it is a 
straightforward, effective, and efficient method. We were able to attain state- 
of-the-art outcomes by efficiently utilizing the resourced dataset. 
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1. INTRODUCTION 

Colon cancer is one of the leading causes of death worldwide. According to the World Health 
Organization (WHO), there were estimated 1,849,519 new cases of colon cancer and 880,792 related deaths in 
2018. Colorectal cancer is one of the most common cancers worldwide and one of the leading causes of cancer 
deaths [1]. In a study by the American Cancer Society, colorectal cancer was found to be one of the most 
common causes of cancer deaths in men and women. Therefore, the diagnosis of colon cancer becomes very 
important, especially in the early stages [2]. A reliable method to ensure that cancer is present in all parts of 
the body [3], [4]. Literature in the area of medical science has recorded evidence of this. However, there are 
only a few papers describing the most recent deep learning optimization accolades for histopathology images 
[5], [6]. Pathologists and clinical researchers have recognized and contributed to the use of histopathological 
imaging to detect tumours. However, this technique can be replaced by the new deep learning methods, for 
certain reasons which included traditional biopsy methods for diagnosing colon cancer are time-consuming 
and they require the prior knowledge of technicians, and the results may vary [7], [8]. Work experience of the 
pathologist is a crucial factor in the traditional diagnosis methods [9], [10]. Due to the recent advances in neural 
networks and image classification, convolutional neural networks (CNN) are now used to analyze 
histopathological images and classify cancer tissues [11], [12]. CNN has been a defining factor for classifying 
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images and making decision in last few years [13], [14]. Since 2012 to 2019, when the ground breaking 
Alexnet, ImageNet, VGG16 and Resnet50, and other models [15], [16]. CNN has contributed to many advances 
in computer vision and image analysis, and histopathological imaging is one of the most important deep 
learning applications in the medical field [17], [18]. Much progress has been made in histopathological 
imaging, but there are not many studies on the classification of colorectal cancer imaging. Furthermore, many 
results of various studies still have unreliable accuracy results [19], [20]. In this paper, we aim to address the 
gap in colorectal cancer imaging in the existing review of literature. We used advanced deep learning CNN 
architectures such as Resnet152, Resnet50, and Vgg16 [21], [22]. A novel deep learning architecture with fine- 
tuning and image augmentation technique has been proposed to classify histopathological images, we used 
multi-class colorectal cancer image dataset. The proposed method experimented on the acquired dataset from 
the Kaggle database to evaluate the results. 

The paper is organized is being as: section 2 contains the discussion on image augmentation technique, 
modified Resnet50, and Adaptive Resnet152. The augmented dataset showed good accuracy and low error rate 
after using the modified Resnet50 and adaptive Resnet152, section 3 includes results and discussion, we 
compared the results with different deep learning models. 


2. PROPOSED METHOD 

Resnet is one of the CNN architectures for large data sets. Compared with other types of neural 
networks, it still needs better competitive performance in classification [23]. The training method uses 
convolution neural network architecture [24], we use predictive model architecture, our goal is to improve 
performance by reducing bias and variance. This method effectively combines the predictions of each 
architecture involved in the evaluation to produce predictions. Robust CNN architectures [25], such as 
Resnet152, Resnet50, and VGG16 can be processed in a fully connected neural network to produce progressive 
results. The image augmentation method as shown in Figure 1, consists of image batches obtained from a 
separate neural network. According to the index, the trained CNN architecture is assigned non-linear weights, 
and the CNN model will be trained. Except for the fully connected layer and the dense layer, the CNN model 
is retrained with the last 10 layers frozen. Then use the colorectal histopathology dataset to train, validate, and 
test the final architecture.We proposed a novel CNN architecture to identify the lowest error rate and loss 
parameters. On the basis of performance indicators, the module was evaluated on measures such as accuracy, 
precision and recovery rate. 


MARE Original Image 


batches of augmentation 
dataset 8 


images object 


Train CNN on Randomly 
batch transformed 
batch of images 


Figure 1. Basic architecture for image augmentation technique 


This approach has been implemented when the imbalance of the upper class is an important factor; however, 
we use this data set to distribute the images class wise, reduce the performance complexity of a given 
architecture, and effectively optimize the model. CNN implementation: 

We apply the convolution technique using the following equation: 


G [M, N]=(F*H) [M, NJ=Xj Xx HL, kIFIM — j, N — K] (1) 
Where, input image is F, filter is H, M and N are represented indexes of rows and columns. 
Resnet 


It is two kinds of residual connections like F(x) and F(x)+x. 


Y=F (x, {Wi})+x (2) 
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Y=F (x, {(Wi)+W,;x (3) 


The first equation is used when the both the input and output dimensions are same. The second 
equation is used when the both the dimensions are not same. 


Zi =D wy ay + b (4) 


where a}! = yl (2!") 


In case where the skip connection is on, we have: 
ZA Ypa wal 4 pf (5) 


where a}! = pl (aP + aa) 

With the right choice of the activation function pw! =0 and bH! = 0 

We can have a! = ali~? the residual block is capable of learning the identity function and thus it does not harm 
the neural network. We usually need a! and a''-2] to have the same shape so we use same convolution. If not, 
we seta’ = plz! + w,a'-?) and dim (w,) = [n',n'~?]. Where W, might be a fixed tensor or learned one. 


2.1. Dataset 

Colorectal histopathology images dataset is available in Kaggle.com. Submitted by the Kather et al., 
2019 [26], open source, including compressed files, composed of 5000 images, each image is 224x224 pixels, 
each image is reduced to 64x64 and eight classes are evenly distributed in non-overlapping categories such as 
tumour, stroma, complex, lympho, debris, mucosa, adipose, and empty. We had increased all the eight classes 
of generated images equally by using image augmentation technique. This dataset consists of total 20000 
images and has multi-class tissue characteristics. Now, the 20000 colorectal histopathology images are splitted 
into 60% for training, 10% for validation and 30% for testing. 


2.2. Modified Resnet50 

Residual network (ResNet) is pretrained model. It is first proposed the concept of skip the connection. 
Figure 2 shows the transition connection as well as the stacking of convolution layers one after the other. In 
addition to the conventional stacking of the convolutional layer we also added the original input to the output 
of the convolutional block. The ResNet50 model is a pre-trained neural network, it has 5 stages, with a 
convolution block and an i- identification block. In Figure 2, each convolution block has 3 convolution layers, 
and each identification block also has 3 convolution layers ranging up to 23 million trainable parameters. We 
proposed this model for Colorectal cancer classification. Our proposed pretrained Resnet50 model uses freeze 
layers to add the output from an earlier layer to a later layer. 


STAGE 1, STAGE 2 and STAGE 3 are Freeze layers 


Input image STAGE 1: STAGE 2: STAGE 3: 
224 x 224x3 Convi:7x7 Conv3:3x3 Conv4:3x3 
O/P SIZE:112 x 112 O/P SIZE: 56 x 56 O/P SIZE: 28 x 28 


Fully Connected Fully Connected STAGE 5: STAGE 4: 
O/P SIZE: 1x1 O/P SIZE:1x1 Convé6: 3x3 Convé6: 3x3 
512x8 2048 x 512 O/P SIZE: 7 x7 O/P SIZE: 14 x 14 


Out put layer 
Softmax:8 class 


Figure 2. Architecture of Resnet50 freeze layer 
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2.3. Adaptive Resnet152 

Resnet152 is 152 layers pretrained model. Which introduces the concept of residual learning, 
generally used for random access connections to learn feature subtraction at the input of this level. Residual 
training has proven to improve the performance of model training. Resnet152 has over 23 million parameters. 
Our proposed pretrained Resnet152 model uses the input image 224x224 pixels it passed to 6 convolution 
blocks. Each block had a filter size 3x3 with stride value 2 i.e., Figure 3. Resnet152 produces low error rate 
and it improve the model performance. We used the same dataset for Resnet152 architecture model. Which 
was executed with colorectal cancer image dataset with 60% training, and 30% testing and 10% for validation. 


Convi Pool 3x Conv2_x 


Patch :7x7 Patch :3x3 Conv: 1x1,64 


224 x 224 x3 ; ; 
% ci4x Stride: 2 Stride :2 Conv: 3x3,64 
Conv: 1x1.256 


Input image 


3x Conv6_x 36 x Conv5_x 8x Conv4_x 8x Conv3_x 

Conv: 1x1,1024 Conv: 1x1,512 Conv: 1x1,256 Conv: 1x1,128 
Conv: 3x3,1024 Conv: 3x3,512 Conv: 3x3,256 Conv: 3x3,128 
Conv: 1x1,4096 Conv: 1x1.2048 Conv: 1x1,1024 Conv: 1x1,512 


Figure 3. Basic architecture of Resnet152 with 6 convolution blocks 


3. RESULTS AND DISCUSSION 

In this research, we experimented on 8 models, each model is tested on the same dataset. However, 
we reconstructed and demonstrated the performances of individual model. It can be visualized for the eight 
classes of colorectal cancer images which is shown in Figure 4. The eight classes are Tumor, Stroma, Complex, 
Lympho, Debris, Mucosa, Adipose, and Empty. 
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Figure 4. Representation of 8-multiclass colorectal histopathology images 


In our trained CNN design, the trained model was divided into 2 major parts. In the 1st phase we have 
a tendency to trained individual architectures with pretrained individual models like Resnet152, Resnet50, 
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VGG16, and the configurable architecture of the CNN model. In the second phase, we checked the results 
obtained from the creation of a single model and a CNN. The performance of our proposed method is compared 
with the existing methods. 

The performance analysis of eight individual models is shown in Figure 5. Finally, we compared 
results that had a classification of test accuracy of more than 90%. Individual performance accuracy for 
Resnet152, Resnet50, and VGG16 is 98.38%, 97.08%, and 96.14%, respectively, and the images are accurately 
identified. The comparative results are shown in Table 1. The proposed adaptive Resnet152 achieved better 
accuracy rate of 98.38% and lowest error rate compared to existing models. These results show the 
effectiveness of the proposed method. 
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Figure 5. Graphical representation of comparative results 


Table 1. Comparative results of histopathology cancer images multi-class classification 


S.NO. No. of Epochs Models Training accuracy _ Validation accuracy __ Testing accuracy 
1 30 Adaptive Resnet152 99.65 97.53 98.38 
2 30 Modified Resnet50 99.05 97.62 97.08 
3 30 Resnet50 98.22 94.55 96.89 
4 30 Neural Net 98.53 96.13 96.34 
5 30 Veggl6 98.43 95.83 96.14 
6 30 ImageNet 97.55 95.02 95.98 
7 30 Vgg19 95.35 94.10 95.53 
8 30 AlexNet 95.75 93.2 94.45 


4. CONCLUSION 

In this paper, we proposed a deep conventional neural network model for multi class classification of 
colorectal histopathology images. The last ten layers of network are feezed and re-trained the main neural 
network with dense layer. We successfully improved the accuracy for classification of multi class colorectal 
histopathology images. Through adaptation of the Resnet152 and reconstruction of Resnet50 models, we 
trained the CNN architecture for best classification accuracy 98.38% and lowest error rate. These results lead 
to reliable disease identification tumour from colorectal histopathology images. The results acquired were 
arguably more sophisticated than the currently existing models. In the future work, we are planning to use 
transfer learning method to obtain better results. 
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